Implementation and test of activation-verification mechanisms

and embedding in ACORNS We describe a bottom-up, activation-based paradigm for continuous speech recognition. Speech is represented by co-occurrence statistics of acoustic events over an analysis window of variable length, leading to a vector representation of high but fixed dimension called “Histogram of Acoustic Cooccurrence” (HAC). During training, recurring acoustic patterns are discovered and associated to words through non-negative matrix factorization (NMF). During testing, word activations are computed from the HAC-representation and their time of occurrence is estimated. Hence, words in a continuous utterance can be detected, located and ordered. The plausibility of word activations is verified in multiple ways. First, the activations must exceed a threshold. Second, the locations (in time) of the detected words needs to be consistent over time. Third, we verify if the order in which words are activated corresponds to the expected activation patterns as learned through previous exposure to the language, a mechanism which is implemented as a second NMF-based learning and detection layer. With this deliverable, we show that it is possible to acquire a vocabulary of 51 words using under supervision consisting of unordered utterance-level tags that simulate the multimodal input. Refer to D5.4.2 for experiments with weaker forms of supervision. Verification of detected words against knowledge sources such as a language model is important for scaling the vocabulary further towards the final goals of the project. The way in which we achieve this – through a second NMF layer – also proves that NMF-based learning can be cascaded, a technique that we want to apply at the lower, acoustic, level of speech processing. By introducing a layer below the current ones, reusability of acoustic representations and scalability towards a larger vocabulary size will be shown in the final year of ACORNS.